Libp2p stream limits #1017

nazar-pc · 2022-12-12T16:09:33Z

This implements stream libp2p stream limits as described in https://www.notion.so/subspacelabs/libp2p-stream-limits-b787bd71ce6f400d95e781c69932d112

There is some extra minor refactoring in separate commits, but overall the changes are these:

subspace-networking has more conservative stream limits (comparing to defaults)
different tasks launched against networking use part of limited networking capacity for requests and too many requests will be queued
now unnecessary batching restrictions and corresponding CLI parameters were removed
capacity is somewhat elastic and scaled with number connected peers

NOTE: It is expected that you'll see this message until other nodes are updated:

dropping (Stream 41d73bd4/8040) because buffer is full

Code contributor checklist:

I have read, understood and followed contributing guide

…e default

rahulksnv · 2022-12-12T19:22:10Z

crates/subspace-networking/src/node.rs

@@ -272,6 +311,18 @@ impl Node {
        &self,
        key: Multihash,
    ) -> Result<impl Stream<Item = Vec<u8>>, GetValueError> {
+        let kademlia_permit = self


Can the shared::Command be split into something like shared::{KademliaCommand, GlobalCommand}, and the corresponding async task in node_runner::handle_command() also be split? This way, the semaphores also can be separated, and avoid having to take both of them for kademlia specific tasks

Kademlia limits are a subset of global limits, hence I don't think we want to split them. Unless we subtract Kademlia limits from global, which is actually an interesting and probably good idea, but even then I don't see how splitting commands will help as the bottleneck we're considering here is connection level streams in libp2p that are shared between different behavior, including Kademlia.

Ah I see, yeah in that case splitting doesn't help.

Just that some paths grabbing 1 sem, others grabbing 2 makes it bit harder to reason about. Would be great if these can be unified

rahulksnv · 2022-12-12T20:15:46Z

crates/subspace-networking/src/create.rs

+const GLOBAL_CONCURRENT_TASKS_BOOST_PER_PEER: usize = 2;
+const SEMAPHORE_MAINTENANCE_INTERVAL: Duration = Duration::from_secs(5);
+
+async fn maintain_semaphore_permits_capacity(


Guess semaphores are not best fit for this scenario, where the max limit can grow/shrink as peers come and go. Hence we need this complexity to hold excess permits, etc

How do you feel about this:

Maintain current_max_limit, current_usage. These two are protected by async Mutex/CV.

current_max_limit can be updated with the boost_per_peer from handle_swarm_event() during connection open/close.

Senders: wait for permits to be freed, add to current_usage

From drop() of the permit: bump up current_usage, signal any waiters. But, any excess permits if the max limit shrinked are not returned to the free pool. This would need spawning async task from the dtor unfortunately, as rust doesn't support await from drop yet.

Happy to make a branch off your PR and make a draft, if that helps (also to get my feet weet)

It is a bit hacky, I agree, but not too much (for my taste).

I found CVs counter-intuitive to read in most cases and I feel like we'll implement essentially the same mechanism (practically speaking) in the end. It might be an interesting exercise, but I'm not sure it'll actually be a lot of value in that.

Supporting what we need here (resizing, dropping of unfinished acquisition) is probably quire a bit of effort.

Feel free to try unless you have anything more valuable to do.

OK, looks like tokio doesn't have CondVar, so that won't work :-( But found sem has a forget() API: https://docs.rs/tokio/1.23.0/tokio/sync/struct.SemaphorePermit.html#method.forget

This can help avoid the reserved_permits stash, which would simplify this quite a bit:

if total_permits < expected_total_permits { semaphore.add_permits(expected_total_permits - total_permits); } else { let num_excess = total_permits - expected_total_permits; // acquire()/forget() num_excess permits }

Not a big deal: it may take a while for the changes to take effect (in the shrink case), as the sem acquisition in this path doesn't get higher priority (unless done from the dtor as in the other approach)

We can use forget, but that'll mean we'll permanently increase internal capacity of the semaphore every time we call forget. And while we'll probably never reach https://docs.rs/tokio/1.23.0/tokio/sync/struct.Semaphore.html#associatedconstant.MAX_PERMITS on 64-bit platform, we can on 32-bit platform. It'll take a while, but when you hit it, it'll be hard to debug.

yeah you are right.

I still feel this path can be simplified (took a bit to figure out this shrinking business), let me try out the idea I mentioned above off your branch.

i1i1 · 2022-12-13T04:25:35Z

crates/subspace-networking/src/node_runner.rs

+                let shared = match self.shared_weak.upgrade() {
+                    Some(shared) => shared,
+                    None => {
+                        return;
+                    }
+                };


Just a note. We can do this right now:

let Some(shared) = self.shared_weak.upgrade() else { return };

That is exactly what I wrote initially. Unfortunately rustfmt seems to be unable to format let .. else, which is why I'm not using it yet anywhere. I'll be checking with new Rust releases as we upgrade and will start using it once formatting works.

shamil-gadelshin

Good work! It will significantly simplify concurrency management. However, I have a couple of major concerns:

maintain_semaphore_permits_capacity requires testing with edge cases
all Node methods should be protected from incorrect semaphore.acquire_permit ordering by introducing wrapping-methods
WrapperWithPermits uses Vec for semaphore permits which doesn't guarantee drop order.

shamil-gadelshin · 2022-12-13T04:58:50Z

crates/subspace-networking/src/create.rs

+    boost_per_peer: usize,
+) {
+    let base_permits = semaphore.available_permits();
+    // Total permits technically supported by semaphore


This method is critical for maintaining concurrent requests feature and despite extensive comments is still hard to read. I recommend converting it to the testable version and applying several tests including edge cases.

Made maintenance interval an argument and wrote some tests

shamil-gadelshin · 2022-12-13T05:01:39Z

crates/subspace-networking/src/node.rs

    }

    pub async fn put_value(
        &self,
        key: Multihash,
        value: Vec<u8>,
    ) -> Result<impl Stream<Item = ()>, PutValueError> {
+        let kademlia_permit = self


The order of the semaphore access is important. With some combination of the permit number, we could experience race conditions. Let's have two wrapper methods for all kademlia- and non-kademlia-methods with the fixed order of acquiring resources. The wrapper method should acquire semaphores and return a WrapperWithPermits for the inner command.

This will also reduce similar copy-paste code in the methods.

I made two semaphores non-overlapping, so you only need to take one and not both. Hope it helps. And you are right about ordering, it can be a source of issues.

crates/subspace-networking/src/node.rs

…emaphore needs to be taken

nazar-pc

Added connected_peers-1, which was missing, some tests and made semaphores non-overlapping, inspired by Rahul's comments.

nazar-pc · 2022-12-13T09:42:11Z

crates/subspace-networking/src/create.rs

+    boost_per_peer: usize,
+) {
+    let base_permits = semaphore.available_permits();
+    // Total permits technically supported by semaphore


Made maintenance interval an argument and wrote some tests

nazar-pc · 2022-12-13T09:43:16Z

crates/subspace-networking/src/node.rs

    }

    pub async fn put_value(
        &self,
        key: Multihash,
        value: Vec<u8>,
    ) -> Result<impl Stream<Item = ()>, PutValueError> {
+        let kademlia_permit = self


I made two semaphores non-overlapping, so you only need to take one and not both. Hope it helps. And you are right about ordering, it can be a source of issues.

nazar-pc · 2022-12-13T09:44:09Z

crates/subspace-networking/src/node_runner.rs

+                let shared = match self.shared_weak.upgrade() {
+                    Some(shared) => shared,
+                    None => {
+                        return;
+                    }
+                };


That is exactly what I wrote initially. Unfortunately rustfmt seems to be unable to format let .. else, which is why I'm not using it yet anywhere. I'll be checking with new Rust releases as we upgrade and will start using it once formatting works.

shamil-gadelshin

Looks good but something is wrong:

shamix@shamix2 subspace % RUST_LOG=info ./target/debug/subspace-node --chain dev --tmp --execution wasm --pruning 1024 --keep-blocks 1024 --validator --force-authoring --rpc-max-subscriptions-per-connection=100000 --ws-max-connections=100000  --dsn-bootstrap-nodes=/ip4/127.0.0.1/tcp/50000/p2p/12D3KooWBTzCDwwRtkUz8unUFUDMP9wEhEDNPfNH6bsuhbz4RrRr
2022-12-13 16:58:37 Subspace    
2022-12-13 16:58:37 ✌️  version 0.1.0-b43bd6caa6d    
2022-12-13 16:58:37 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022    
2022-12-13 16:58:37 📋 Chain specification: Subspace development    
2022-12-13 16:58:37 🏷  Node name: amuck-truck-1899    
2022-12-13 16:58:37 👤 Role: AUTHORITY    
2022-12-13 16:58:37 💾 Database: ParityDb at /var/folders/sh/t2v54wn12vbbdlt7_ytgkxbh0000gn/T/substratedxFRHX/chains/subspace_dev/paritydb/full    
2022-12-13 16:58:37 ⛓  Native runtime: subspace-1 (subspace-0.tx0.au0)    
2022-12-13 16:58:38 [PrimaryChain] 🔨 Initializing Genesis block/state (state: 0xf521…0b46, header-hash: 0x26c0…4be4)    
2022-12-13 16:58:39 [DSN] DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWHQ81eUgapK2VzxNJReU4o46tL6chwBzMWFvCiyW2mcQe
2022-12-13 16:58:39 [PrimaryChain] Subspace networking initialized: Node ID is 12D3KooWHQ81eUgapK2VzxNJReU4o46tL6chwBzMWFvCiyW2mcQe
2022-12-13 16:58:39 [PrimaryChain] Starting archiving from genesis    
2022-12-13 16:58:39 [PrimaryChain] Archiving already produced blocks 0..=0    
2022-12-13 16:58:40 [PrimaryChain] Using default protocol ID "sup" because none is configured in the chain specs    
2022-12-13 16:58:40 [PrimaryChain] Processing a segment. segment_index=0
2022-12-13 16:58:40 [PrimaryChain] 🏷  Local node identity is: 12D3KooWHQ81eUgapK2VzxNJReU4o46tL6chwBzMWFvCiyW2mcQe    
2022-12-13 16:58:40 [PrimaryChain] 🧑‍🌾 Starting Subspace Authorship worker    
2022-12-13 16:58:40 [PrimaryChain] 💻 Operating system: macos    
2022-12-13 16:58:40 [PrimaryChain] 💻 CPU architecture: aarch64    
2022-12-13 16:58:40 [PrimaryChain] 📦 Highest known block at #0    
2022-12-13 16:58:40 [PrimaryChain] 〽️ Prometheus exporter started at 127.0.0.1:9615    
2022-12-13 16:58:40 [PrimaryChain] Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=Some(["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"])    
2022-12-13 16:58:40 [PrimaryChain] Running JSON-RPC WS server: addr=127.0.0.1:9944, allowed origins=Some(["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"])    
2022-12-13 16:58:40 [PrimaryChain] Piece publishing was successful. segment_index=0
2022-12-13 16:58:45 [PrimaryChain] 💤 Idle (0 peers), best: #0 (0x26c0…4be4), finalized #0 (0x26c0…4be4), ⬇ 0 ⬆ 0    
2022-12-13 16:58:50 [PrimaryChain] 💤 Idle (0 peers), best: #0 (0x26c0…4be4), finalized #0 (0x26c0…4be4), ⬇ 0 ⬆ 0    
2022-12-13 16:58:55 Accepting new connection 1/100000
2022-12-13 16:58:55 Accepting new connection 2/100000
2022-12-13 16:58:55 [PrimaryChain] 💤 Idle (0 peers), best: #0 (0x26c0…4be4), finalized #0 (0x26c0…4be4), ⬇ 0 ⬆ 0    
2022-12-13 16:59:00 [PrimaryChain] 💤 Idle (0 peers), best: #0 (0x26c0…4be4), finalized #0 (0x26c0…4be4), ⬇ 0 ⬆ 0    
2022-12-13 16:59:05 [PrimaryChain] 💤 Idle (0 peers), best: #0 (0x26c0…4be4), finalized #0 (0x26c0…4be4), ⬇ 0 ⬆ 0    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. No older substream waiting to be reused. Dropping new substream.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.    
2022-12-13 16:59:10 New inbound substream to PeerId("12D3KooWLnjgzSammRdGG5vr6vSyfgauVhuvkNtKdgH1a4sqkJSW") exceeds inbound substream limit. Removed older substream waiting to be reused.

crates/subspace-networking/src/create/tests.rs

…gradual concurrency scaling

nazar-pc · 2022-12-14T22:48:54Z

Pushed two important updates, but PR is still in draft state, the reason is primarily puts that seem to be causing exceeding of incoming stream limits in Kademlia. Upstream discussion: libp2p/rust-libp2p#3235

* Add ResizableSemaphore * Clean ups * Add unit tests * Use the sem changes * Clean up * Fix race * Remove async lock * Restructure * Address comments * Bring back threshold for bumping quota * Address comments * Fix expect message Co-authored-by: Nazar Mokrynskyi <[email protected]>

…ability of snowball effect due to pieces not published in time

nazar-pc · 2022-12-28T09:21:44Z

Updated libp2p to upstream git version first to separate changes, then to our fork that contains just libp2p/rust-libp2p#3287 to fix concurrency issues we were hitting before.

Since pieces are relatively large and there are many of them, 5 seconds often wasn't enough to finish requests, so I increased it to make sure we can finish publishing of the piece before trying again.

Overall, seems to work well and ready for review again.

nazar-pc · 2022-12-28T16:21:40Z

I have done testing locally against Gemini 3b and it looks promising, merging

nazar-pc added 5 commits December 12, 2022 16:17

Remove setting yamux window update mode, the same value is already th…

6b3b68b

…e default

Restore tracking of number of connected peers

7d1247f

Implement libp2p stream limits with wrappers

bfac7a9

Remove now unnecessary batching restrictions

d50c0a9

Remove unnecessary boxing and pinning in various places

acebfaa

nazar-pc requested review from liuchengxu, shamil-gadelshin and rahulksnv December 12, 2022 16:09

nazar-pc requested review from rg3l3dr, i1i1 and vedhavyas as code owners December 12, 2022 16:09

rahulksnv reviewed Dec 12, 2022

View reviewed changes

i1i1 previously approved these changes Dec 13, 2022

View reviewed changes

shamil-gadelshin suggested changes Dec 13, 2022

View reviewed changes

nazar-pc added 2 commits December 13, 2022 10:48

Subtract Kademlia tasks concurrency from the rest, so that only one S…

5029bf8

…emaphore needs to be taken

Write basic tests for maintain_semaphore_permits_capacity

b43bd6c

nazar-pc dismissed i1i1’s stale review via b43bd6c December 13, 2022 09:41

nazar-pc commented Dec 13, 2022

View reviewed changes

nazar-pc requested a review from shamil-gadelshin December 13, 2022 09:46

shamil-gadelshin suggested changes Dec 13, 2022

View reviewed changes

i1i1 reviewed Dec 13, 2022

View reviewed changes

crates/subspace-networking/src/create/tests.rs Show resolved Hide resolved

i1i1 previously approved these changes Dec 13, 2022

View reviewed changes

nazar-pc marked this pull request as draft December 13, 2022 21:47

rahulksnv mentioned this pull request Dec 13, 2022

Resizable semaphore #1019

Merged

1 task

nazar-pc added 2 commits December 15, 2022 00:17

Drop permit when query is finished and not earlier

6060dd6

Support peers boosting threshold customization, set it to 5 for more …

fc80120

…gradual concurrency scaling

nazar-pc dismissed i1i1’s stale review via fc80120 December 14, 2022 22:47

nazar-pc added 4 commits December 27, 2022 22:13

Merge remote-tracking branch 'origin/main' into libp2p-stream-limits

ce20efc

Update to newer libp2p revision

47fb5db

Update libp2p to fork with kademlia handler buffering

e98c358

Increase timeout and max interval for piece publishing to reduce prob…

21e719c

…ability of snowball effect due to pieces not published in time

nazar-pc marked this pull request as ready for review December 28, 2022 09:21

nazar-pc requested review from rahulksnv, i1i1 and shamil-gadelshin December 28, 2022 09:21

i1i1 approved these changes Dec 28, 2022

View reviewed changes

nazar-pc merged commit 3396465 into main Dec 28, 2022

nazar-pc deleted the libp2p-stream-limits branch December 28, 2022 16:21

nazar-pc mentioned this pull request Jan 4, 2023

DSN. Make batching semaphore global for the farmer. #996

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Libp2p stream limits #1017

Libp2p stream limits #1017

nazar-pc commented Dec 12, 2022 •

edited

Loading

rahulksnv Dec 12, 2022

nazar-pc Dec 12, 2022

rahulksnv Dec 12, 2022

rahulksnv Dec 12, 2022 •

edited

Loading

nazar-pc Dec 12, 2022

rahulksnv Dec 12, 2022

nazar-pc Dec 13, 2022

rahulksnv Dec 13, 2022

i1i1 Dec 13, 2022

nazar-pc Dec 13, 2022

shamil-gadelshin left a comment

shamil-gadelshin Dec 13, 2022

nazar-pc Dec 13, 2022

shamil-gadelshin Dec 13, 2022

shamil-gadelshin Dec 13, 2022

nazar-pc Dec 13, 2022

nazar-pc left a comment

nazar-pc Dec 13, 2022

nazar-pc Dec 13, 2022

nazar-pc Dec 13, 2022

shamil-gadelshin left a comment •

edited by nazar-pc

Loading

nazar-pc commented Dec 14, 2022

nazar-pc commented Dec 28, 2022

nazar-pc commented Dec 28, 2022

Libp2p stream limits #1017

Libp2p stream limits #1017

Conversation

nazar-pc commented Dec 12, 2022 • edited Loading

Code contributor checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahulksnv Dec 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shamil-gadelshin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nazar-pc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shamil-gadelshin left a comment • edited by nazar-pc Loading

Choose a reason for hiding this comment

nazar-pc commented Dec 14, 2022

nazar-pc commented Dec 28, 2022

nazar-pc commented Dec 28, 2022

nazar-pc commented Dec 12, 2022 •

edited

Loading

rahulksnv Dec 12, 2022 •

edited

Loading

shamil-gadelshin left a comment •

edited by nazar-pc

Loading